[core] Add option to support reading sequence_number in AuditLogTable and BinlogTable #6933

wzhero1 · 2025-12-30T11:07:33Z

Purpose

This PR adds support for reading _SEQUENCE_NUMBER field in audit_log and binlog system tables for primary key tables.

A new configuration option changelog-read.sequence-number.enabled (default: false) is introduced to control whether to include the _SEQUENCE_NUMBER field in the output schema of audit_log and binlog system tables. When enabled, users can access the internal sequence number that Paimon uses for ordering records within primary key tables.

Key changes:

Added new CoreOptions.CHANGELOG_READ_SEQUENCE_NUMBER_ENABLED configuration
Modified AuditLogTable and BinlogTable to optionally include _SEQUENCE_NUMBER field in their row types
Updated ValueContentRowDataRecordIterator to output sequence number when enabled
Added schema validation to ensure this option is only valid for primary key tables
Updated predicate conversion logic to handle the additional system field

Tests

AuditLogTableTest - basic auditlog table reading and sequence number with table option
BinlogTableTest - basic binlog table reading and sequence number with table option
BatchFileStoreITCase - Flink integration test for audit_log and binlog reading sequence number

API and Format

New configuration option: changelog-read.sequence-number.enabled (Boolean, default: false)
When enabled, audit_log and binlog system tables will have an additional _SEQUENCE_NUMBER column after rowkind
No storage format changes

Documentation

Configuration documentation updated in core_configuration.html

… and BinlogTable

yunfengzhou-hub

Thanks for the PR. Left some comments as below.

yunfengzhou-hub · 2026-01-07T08:48:53Z

docs/layouts/shortcodes/generated/core_configuration.html

+            <td><h5>changelog-read.sequence-number.enabled</h5></td>
+            <td style="word-wrap: break-word;">false</td>
+            <td>Boolean</td>
+            <td>Whether to include _SEQUENCE_NUMBER field in audit_log and binlog system tables. This is only valid for primary key tables.</td>


The output result of an audit log table can be controlled by incremental-between-scan-mode, where changelog is only one of the options. So we might better not include the word "changelog" in the config option name.

Changed to table-read.sequence-number.enabled for clearer semantics, though I'm not sure if this will be supported on regular tables in the future. If it is, the scope of the parameter would become limited.

yunfengzhou-hub · 2026-01-09T08:49:08Z

paimon-core/src/main/java/org/apache/paimon/table/system/AuditLogTable.java


    public AuditLogTable(FileStoreTable wrapped) {
        this.wrapped = wrapped;
+        this.wrapped.schema().options().put(AUDIT_LOG_ENABLED, "true");


How about organize the two new configurations in the following way?

User sets config A.

In AuditLogTable, the code detects whether config A is set. If so, set config B to true.

In ValueContentRowDataRecordIterator, the code only checks config B, instead of configA && configB.

The benefit of such structure is that it can decouple system table concepts from ValueContentRowDataRecordIterator. config B need not be related to audit log table.

Done. The parameters are now aggregated and updated in AuditLogTable, and ValueContentRowDataRecordIterator is only aware of the addSequenceNumberFirst behavior.

The current parameter is AUDIT_ADD_SEQUENCE_NUMBER_FIRST, which clearly reflects what it does (the naming can be discussed further, but it's an internal parameter so it's easy to adjust).

yunfengzhou-hub · 2026-01-09T08:52:43Z

paimon-core/src/main/java/org/apache/paimon/table/system/AuditLogTable.java


-    private final FileStoreTable wrapped;
+    /** Number of special fields (rowkind, and optionally _SEQUENCE_NUMBER). */
+    protected final int specialFieldCount;


How about introducing List<SpecialField> specialFields? If so, we will be able to change

fields.add(SpecialFields.ROW_KIND); if (specialFieldCount > 1) { fields.add(SpecialFields.SEQUENCE_NUMBER); }

into

fields.add(specialFields);

changed to List<SpecialField> specialFields

yunfengzhou-hub · 2026-01-09T08:55:14Z

paimon-core/src/main/java/org/apache/paimon/table/system/AuditLogTable.java

+                // _SEQUENCE_NUMBER is at index 0 in bottom output
+                return row.getLong(0);
+            }
+            return super.getLong(pos);


If _SEQUENCE_NUMBER is at index 0, then the other fields should be acquired through pos - 1.

The mapping logic is implemented in AuditLogRead.defaultProjection() (for select *) and AuditLogRead.buildProjection(RowType readType) (for selecting specific fields).

yunfengzhou-hub · 2026-01-09T09:05:09Z

paimon-core/src/main/java/org/apache/paimon/table/system/AuditLogTable.java

+        public long getLong(int pos) {
+            int index = indexMapping[pos];
+            if (index == AuditLogRead.SEQUENCE_NUMBER_INDEX) {
+                // _SEQUENCE_NUMBER is at index 0 in bottom output


The reason behind this comment is located in another class(ValueContentRowDataRecordIterator), and these two classes are mainly associated with AUDIT_LOG_ENABLED, which shows a generic concept not related to sequence number.

Thus it might increase the other developer's burden to understand why "_SEQUENCE_NUMBER is at index 0". We might need to think about how to increase code readability here.

I've optimized the _SEQUENCE_NUMBER index logic in defaultProjection() and buildProjection(RowType readType), so getLong no longer needs special handling here.

yunfengzhou-hub · 2026-01-09T09:12:51Z

paimon-core/src/test/java/org/apache/paimon/table/system/AuditLogTableTest.java

        write(table, GenericRow.ofKind(RowKind.INSERT, 1, 2, 5));
        write(table, GenericRow.ofKind(RowKind.UPDATE_BEFORE, 1, 2, 5));
-        write(table, GenericRow.ofKind(RowKind.UPDATE_AFTER, 1, 4, 6));
+        write(table, GenericRow.ofKind(RowKind.UPDATE_AFTER, 1, 2, 6));


This change seems unnecessary.

The original test case is incorrect - the primary keys for RowKind.UPDATE_BEFORE and RowKind.UPDATE_AFTER are inconsistent. This scenario should not occur in normal circumstances.

yunfengzhou-hub · 2026-01-09T09:13:34Z

paimon-core/src/test/java/org/apache/paimon/table/system/AuditLogTableTest.java

        expectedRow.add(
                GenericRow.of(
-                        BinaryString.fromString(RowKind.UPDATE_BEFORE.shortString()), 1, 2, 5));
+                        BinaryString.fromString(RowKind.UPDATE_AFTER.shortString()), 1, 2, 6));


In the original code, the expected result contains 4 rows. Here only three rows are left, missing an UPDATE_BEFORE row.

The original test case is incorrect - the primary keys for RowKind.UPDATE_BEFORE and RowKind.UPDATE_AFTER are inconsistent. This scenario should not occur in normal circumstances.

yunfengzhou-hub · 2026-01-09T09:22:54Z

...on-flink/paimon-flink-common/src/test/java/org/apache/paimon/flink/BatchFileStoreITCase.java

+        // Create primary key table with changelog-read.sequence-number.enabled option
+        sql(
+                "CREATE TABLE test_table_seq (a int PRIMARY KEY NOT ENFORCED, b int, c AS a + b) "
+                        + "WITH ('changelog-read.sequence-number.enabled'='true');");


Despite that users are allowed to set changelog-read.sequence-number.enabled when creating or altering table, a more recommended way is to configure it dynamically in the SELECT query through Flink SQL Hints. It should be more suitable for such configurations that are only useful in specific read queries. The usage of SQL hints is like follows

SELECT * FROM `test_table_seq$audit_log`/*+ OPTIONS('changelog-read.sequence-number.enabled' = 'true') */;

Let's add test cases for this use case as well.

Let's also add some test cases to verify what will happen if changelog-read.sequence-number.enabled is configured on append-only tables.

This new parameter cannot be applied to SQL hints, because the rowType passed by the operator is determined before obtaining dynamic options.

Before processing SQL hint parameters, the Flink job first constructs the AuditLogTable to get the rowtype for building upstream/downstream rowtype transmission information.

Later, when reading AuditLogTable, it copies the dynamicOptions.

At this point, the operator's rowtype (without SEQUENCE_NUM before obtaining dynamic parameters) does not match the task's processing rowtype (with SEQUENCE_NUM).

In short, for select *, the operator's rowtype is generated before dynamic options are applied.

I've added dynamicOptions detection to prevent users from accidentally misusing it in SQL hints.

public Table copy(Map<String, String> dynamicOptions) { if (Boolean.parseBoolean( dynamicOptions.getOrDefault( TABLE_READ_SEQUENCE_NUMBER_ENABLED.key(), "false"))) { throw new UnsupportedOperationException( "table-read.sequence-number.enabled is not supported by hint."); } return new AuditLogTable(wrapped.copy(dynamicOptions)); }

[core] Add option to support reading sequence_number in AuditLogTable…

1e104c6

… and BinlogTable

wzhero1 force-pushed the auditlog-support-seqnum-option branch from 8bc109b to 1e104c6 Compare January 8, 2026 11:34

yunfengzhou-hub reviewed Jan 9, 2026

View reviewed changes

wzhero1 force-pushed the auditlog-support-seqnum-option branch from d06c910 to 2259c82 Compare January 12, 2026 09:47

[core] fix1

c2cd7d1

wzhero1 force-pushed the auditlog-support-seqnum-option branch from 2259c82 to c2cd7d1 Compare January 12, 2026 09:50

[core] Add option to support reading sequence_number in AuditLogTable and BinlogTable #6933

Are you sure you want to change the base?

[core] Add option to support reading sequence_number in AuditLogTable and BinlogTable #6933

Uh oh!

Conversation

wzhero1 commented Dec 30, 2025

Purpose

Tests

API and Format

Documentation

Uh oh!

yunfengzhou-hub left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wzhero1 Jan 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wzhero1 Jan 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

wzhero1 Jan 12, 2026 •

edited

Loading

wzhero1 Jan 12, 2026 •

edited

Loading